It is desired to acquire images of arbitrary real-world scenes and then to render new or synthetic images of the scene using a reflectance function representing arbitrary natural illumination in the environment. Such a representation has many applications, such as realistic image synthesis, scene analysis, 3D scanning of cultural artifacts, object and face recognition, and relighting of image-based models.
Typical prior art methods require controlled incident illumination, e.g., either a point light source or a structured illumination pattern, and a large number of illumination images. Systems that use controlled incident illumination are usually large, expensive, and fixed in place in studio, even for scenes with small objects. For large scenes, such as a city, land, or seascape, it is impossible to control the illumination.
Therefore, it is desired to use arbitrary natural illumination and a small number of illumination images to determine a good estimation of reflectance in any scene, no matter the size. In addition, it is desired to provide a small, inexpensive and portable system to do the reflectance function estimation.
The problem of natural scene acquisition is well known in computer graphics and computer vision. In general, a reflectance field is acquired. Reflectance fields are also known as bi-directional scattering-surface reflectance (BSSRDF), Debevec et al., “Acquiring the reflectance field of a human face,” Computer Graphics, SIGGRAPH 2000 Proceedings, pp. 145–156, 2000.
One can also acquire a distribution function, Nicodemus et al., “Geometric considerations and nomenclature for reflectance,” Monograph 160, National Bureau of Standards (US), October 1977. A distribution function is defined as a ratio of incoming to outgoing radiance at two different pixels in a scene, formally, ƒr(ωi,{right arrow over (x)}i,ω0,{right arrow over (x)}0), where ωi is the direction of the incident illumination at a pixel {right arrow over (x)}i, and ω0 is an observation direction of radiance emitted at a pixel {right arrow over (x)}0. That function is eight-dimensional, assuming a two-dimensional parameterization for the points in the scene.
Because sampling of an eight-dimensional function is challenging, a four-dimensional sub-space or 4D slice of the reflectance field can be used.
For larger scenes, it can be assumed that the incident illumination is far away and arrives at pixels in the scene from direction ωi, and the reflectance field is acquired for some fixed viewing direction ω0. The resulting function ƒw(ωi,{right arrow over (x)}0) is called a weighting function or reflectance function of light transport from distant illumination to the observation point, Zongker et al., “Environment matting and compositing,” Computer Graphics, SIGGRAPH 99 Proceedings, pp. 205–214, 1999. However, it is difficult to acquire the 4D reflectance function for arbitrary scenes lit with natural illumination.
Consider a scene where the surface coordinates are discretized into 1000×1000 points, and the incident illumination is represented by a 1000×1000 pixel image. Sampling and tabulating the reflectance function directly requires 1012 values. This poses a challenge for memory storage. Moreover, it is necessary to take over a hundred high-resolution images, making the acquisition time prohibitively long.
In practice, most direct sampling methods use only low-resolution (low-frequency) incident lighting. Consequently, those methods cannot represent high-frequency effects, such as specularities, refraction, and shadows.
On the other hand, environment matting methods focus on reflectance functions for specular, refractive, and glossy materials. However, environment matting methods have difficulty representing hard shadows or a combination of diffuse and specular materials.
Methods for estimating reflectance functions can be classified as forward and inverse methods. Most forward methods sample the reflectance functions exhaustively and tabulate the results. For each incident illumination direction, the forward method stores weights of the reflectance function for a fixed viewing direction. In practice, only low-resolution incident illumination is used, because one reflectance table has to be stored per scene point. Debevec et al. use highest resolution incident illumination with about two thousand directions.
Polynomial texture maps can improve the compactness of the representation by expressing each reflectance field table with a bivariate polynomial, Malzbender et al., “Polynomial texture maps,” Computer Graphics, SIGGRAPH 2001 Proceedings, pp. 519–528, 2001. Those direct methods work very well for diffuse or glossy objects, such as human faces, cultural artifacts, and other objects with complex appearance.
However, those methods cannot represent high-frequency phenomena, such as refractions, light discontinuities, or hard shadows. Furthermore, those methods do not provide progressive refinement of the approximation error with an increasing number of samples.
Wavelet environment matting addresses some of these shortcomings, Peers et al., “Wavelet environment matting,” Proceedings of the Eurographics Symposium on Rendering, Eurographics Association, pp. 157–166, 2003. Images with wavelets patterns are used as incident illumination, and a feedback loop determines the next pattern to emit based on the error of the current approximation. The reflectance function is progressively refined as more measurements are made. However, that method requires many wavelet patterns for highly specular and refractive materials, or scenes with hard shadows, e.g., up to 2400 pattern images. Because each pixel stores coefficients of the corresponding wavelet patterns, that representation becomes rather large, e.g., 2.5 GB per environment matte with lossless compression. Rendering is also time consuming because the rendering requires a weighted sum of many wavelet-basis images of the new incident illumination.
It is also possible to measure six-dimensional slices of the eight-dimensional reflectance field by varying both the position and direction of the incident illumination for a fixed viewpoint, Masselus et al., “Relighting with 4D incident light fields,” ACM Transactions on Graphics (TOG) 22, 3, pp. 613–620, 2003. They use a projector system for scene acquisition and tabulate the results, which limits the sampling resolution.
Inverse methods observe a scene and determine a probability that the observed output came from a particular region in the incident illumination domain. The incident illumination is typically represented by a bounded region, such as an environment map, which is then modeled as a sum of basis functions, e.g., Gaussian kernels, Blinn et al., “Texture and reflection in computer generated images,” Communications of the ACM (SIGGRAPH '76 Proceedings) 19, 10, pp. 542–547, 1976.
The inverse method requires one to determine weights and parameters of basis functions that best explain a given observation, e.g., an image pixel. Environment matting techniques describe the observation in each pixel using only one 2D function. That leads to a very compact representation, and the method works well for specular and refractive objects. That method only works off-line and does not progressively refine approximation errors. Furthermore, only one 2D function is sometimes not expressive enough to capture the properties of complex reflectance functions representing arbitrary scenes under natural illumination, particularly large scenes that can be subject to very complex illumination.
High-quality extensions to environment matting achieve excellent results for specular, refractive, and glossy materials, Chuang et al., “Environment matting extensions: Towards higher accuracy and real-time capture,” Computer Graphics, SIGGRAPH 2000 Proceedings, pp. 121–130, 2000. The box functions are replaced by oriented 2D Gaussian kernels, which can provide a compact representation. However, rendering with complex incident illumination is slow, because the Gaussian kernels have to be convolved with the new illumination. Morevover, that method cannot progressively refine the approximation. In addition, the 2D Gaussian kernels have difficulty representing discontinuous or more complex reflectance functions, such as a small specular lobe inside a larger diffuse component. In addition, that method uses a number of non-linear optimization steps, which can produce instabilities and noise.
Hybrid methods combine a forward method for low-frequency components and high-quality environment matting for high-frequency reflections and refractions. The low-frequency data are compressed using principal component analysis (PCA). However, that representation is not compact, and that method does not address any of the other problems described above.
A pre-computed radiance transport reflectance function can increase the realism in real-time rendering. In such a method, the radiance transport is pre-determined using global illumination algorithms and a detailed model of the scene, Sloan et al., “Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments,” ACM Transaction on Graphics 21, pp. 527–536, ISSN 0730-0301, Proceedings of ACM SIGGRAPH 2002, July 2002.
To improve the performance of the rendering, the incident illumination can be represented using spherical harmonics, or wavelets. In that case, the reflectance function, which is typically stored per vertex as a transfer matrix, can be compressed using PCA, or wavelets, Ng et al., “All-frequency shadows using non-linear wavelet lighting approximation,” ACM Transactions on Graphics 22, pp. 376–381, 2003.
It desired to provide a system and method for estimating reflectance functions for any sized scene that is accurate at all-frequencies, and that deal with many different illumination effects, including illumination discontinuities such as shadows.